The HTTP Client Transport Connectivity Plug-in

About the HTTP client transport

The HTTP client is a transport for use in connectivity plug-ins which can connect to external services over HTTP/REST, perform requests on them and return the response as an event. It can be used by either customizing what codec to use (for example, the JSON codec) and what events to map to, or it can be used using “generic” events and a predefined chain using a JSON codec, where instances are managed via an EPL API and JSON payloads are sent and received. Mapping to events requires more preparation, but gives a powerful type-safe interface for accessing the results and can support more complex mappings and codecs other than JSON, while the generic events allow quick access to simple services over JSON.

The HTTP client transport can encode HTTP requests and decode HTTP responses with gzip or deflate compression format. It also supports HTML form encoding and can encode a dictionary payload to either multipart/form-data or application/x-www-form-urlencoded media types.

When using the event mappings, for each service (host and port combination) that you want to connect to, you must create a new instance of a connectivity chain in your configuration file. To use the service, you send events to that chain, where the events are correctly mapped as described in Mapping events between EPL and HTTP client requests. The response is sent back by the same chain instance, with the configured mapping rules.

This transport does not provide a dynamic chain manager. So chains are created either dynamically from EPL using ConnectivityPlugins.createDynamicChain and a named chain definition specified in the dynamicChains section of the YAML configuration file, or statically using the startChains section of the YAML configuration file. For more information on YAML configuration files, see Using connectivity plug-ins and especially Configuration file for connectivity plug-ins.

Persistent connections to the server are used for multiple requests if this is supported by the service. Connection details to the service are part of the configuration of the transport in the configuration file. Details of the individual requests are configured through the events sent to the chain. The HTTP client supports HTTP version 1.1 and TLS version 1.2 and above.

The HTTP client is designed to talk to REST services and supports GET, POST, PUT and DELETE operations.

Loading the HTTP client transport

You can load the HTTP client transport by adding the HTTP Client connectivity bundle to your project in Apama Plugin for Eclipse (see Adding the HTTP client connectivity plug-in to a project) or using the apama_project tool (see Creating and managing an Apama project from the command line). Alternatively, you can load the transport with the following connectivityPlugins stanza in your YAML configuration file:

connectivityPlugins:
  HTTPClientTransport:
    libraryName: connectivity-http-client
    class: HTTPClient

Configuring the HTTP client transport

The HTTP client should be added to a chain containing the appropriate mapping rules (see Mapping events between EPL and HTTP client requests for detailed information). Connection information is configured through the HTTPClientTransport element in each chain. For example:

startChains:
  HTTPClientChain:
    - apama.eventMap
    *codecs...*
    - HTTPClientTransport:
        host: www.google.com
        basePath: "/myapi/v123"
        port: 80
        timeoutSecs: 120
        tls: false
        tlsAcceptUnrecognizedCertificates: false
        tlsCertificateAuthorityFile: ""
        followRedirects: true
        cookieJar: true
        numClients: 1
        authentication:
          authenticationType: none
          username: ""
          password: ""
        proxy:
          host: ""
          port: ""
          authentication:
            authenticationType: none
            username: ""
            password: ""

The configuration options below can either be configured statically in the configuration file, or via replacement variables. Variables of the form ${varname} are replaced at correlator startup time either from a provided .properties file or from the correlator command line. Variables of the form @{varname} are replaced at chain creation time if using dynamic connections to services (see also Configuring dynamic connections to services).

The following configuration options are available for the HTTP client:

Mapping events between EPL and HTTP client requests

The information in this section applies when you have added the HTTP Client connectivity bundle with the JSON with application-specific event definitions option in Apama Plugin for Eclipse.

The HTTP client accepts requests with metadata fields indicating how to make the request and a binary or dictionary payload to be submitted as the body of the request. Each entry in the dictionary payload should have a string key and either a string or a binary value. If the payload is a dictionary, then metadata.contentType must be set to either multipart/form-data or application/x-www-form-urlencoded. A response contains a binary payload which is the body of the response and further metadata fields describing the response. For the responses to be useful to EPL, they must be converted into the format expected by Apama. This is done using the Classifier codec, Mapper codec and other codecs (see Codec connectivity plug-ins).

In order for EPL to connect a response event to the correct request event, each request contains a top-level requestId field in the metadata. This is returned verbatim in the corresponding response event along with the path and method copied from the request. If these are mapped to or from EPL, then they can be used for a request-response protocol in EPL. For example:

integer id := integer.incrementCounter("HTTPClient.requestId"); // get a
                           // unique ID to differentiate different responses
// listen for success and failure responses
on Response(id=id) as response and not Error(id=id) {
   // handle successful requests
}
on Error(id=id) as error and not Response(id=id) {
   // handle unsuccessful requests
}
send Request(id,.../* more request data here */) to "httpchannel";
   // send the request

The event types used in EPL should be specific to your application and then mapped in the chain to the fields expected by the HTTP client.

The following fields in each event are read by the HTTP client. Field names containing periods (.) indicate nested map structures within the metadata. This nesting is automatically handled by the Mapper codec, and fields can be referred to there just using these names (see also The Mapper codec connectivity plug-in).

You can use the Mapper codec to move things between the payload and the metadata, and vice versa. For example:

startChains:
  HTTPClientChain:
    - apama.eventMap
    - mapperCodec:
        MyRequest:
          towardsTransport:
            mapFrom:
              - metadata.http.path: payload.path
              - metadata.requestId: payload.id
              - payload: payload.body
            defaultValue:
              - metadata.http.method: GET
              - metadata.http.headers.accept: application/json
        MyResponse:
          towardsHost:
            mapFrom:
              - payload.body: payload
              - payload.path: metadata.http.path
              - payload.id: metadata.requestId
        Error:
          towardsHost:
            mapFrom:
              - payload.message: metadata.http.statusReason
              - payload.id: metadata.requestId
              - payload.path: metadata.http.path
              - payload.code: metadata.http.statusCode
    - classifierCodec:
        rules:
          - MyResponse:
              - metadata.http.statusCode: 200
          - Error:
              - metadata.http.statusCode:
    - stringCodec
    - HTTPClientTransport

The above example also demonstrates how to use the Classifier codec to split responses into normal responses and error responses based on the status code (see also Distinguishing response types).

Examples of using the Mapper and Classifier codecs to set these fields can be found in Example mapping rules.

Distinguishing response types

A single chain will often deal with multiple event types in either direction. In the direction towards the transport, the type is already known and can be used to create multiple stanzas in the Mapper codec. For messages towards the host, the event type will not yet have been set. The Classifier codec can use fields in the message (payload or metadata) to set the event type.

For the HTTP client, one of the major distinctions is between success replies and various types of failure. The HTTP status code (metadata.http.statusCode) is used to determine whether or not the response is a success. Typically, a response code of 200 indicates that the request was a success, and anything else would be some kind of error. Both errors returned by the remote host and issues which occur within the client itself are returned as messages with a status code other than 200.

For example, a Classifier codec which wants to just distinguish errors and success would look as follows:

- classifierCodec:
    rules:
      - MyResponse:
          - metadata.http.statusCode: 200
      - Error:
          - metadata.http.statusCode:

There may also be multiple types of success response, possibly from requests to different URLs in the same host. You can use other fields from the metadata or the payload to set the event type. For example:

- classifierCodec:
    rules:
      - LoginSuccess: # OK response with a session cookie set
          - metadata.http.statusCode: 200
          - metadata.http.cookies.session:
      - DataResponse1:
          - metadata.http.statusCode: 200
          - payload.datatype: foo
      - DataResponse2:
          - metadata.http.statusCode: 200
          - metadata.http.path: /data2
      - Error:
          - metadata.http.statusCode:

Handling HTTP headers

The HTTP client reads any number of metadata.http.headers.keyname variables from your event and puts them into the HTTP request. Similarly, any headers returned in the response are mapped to the same variables in the response. Some special handling is applied as described below.

All HTTP headers are converted from ISO-8859-1 (the character set for HTTP headers as defined in the RFC publications) to UTF-8 in the metadata (and vice versa for requests).

All HTTP header keys are converted to lowercase in both directions (since HTTP header keys are defined to be case-insensitive). You should use lowercase in all of your mapping and classification rules.

Any HTTP headers for which multiple values have been provided for a single key (after normalization of case) are dropped in either direction.

The following HTTP headers are handled specially in requests:

In addition, the top-level fields metadata.charset and metadata.contentType are set in the response from the HTTP content-type header.

Cookie and Set-Cookie headers are handled specially. See Dealing with cookies.

Handling HTML form encoding

If the body of the request is a dictionary payload having a string key and either a string or binary value, the request body is then encoded to either multipart/form-data or application/x-www-form-urlencoded media types, depending on metadata.contentType.

If metadata.contentType is set to application/x-www-form-urlencoded, then the dictionary payload must have string keys and string values and is transmitted as URL-encoded form data.

If metadata.contentType is set to multipart/form-data, then the dictionary payload is encoded to multi-part form data. This method must be used to send non-ASCII text or binary data. The binary data form fields should have the following additional metadata: filename, contentType and charset. filename is a required parameter.

You can put these metadata items in a form dictionary as follows:

metadata.http.form.*name*.contentType
metadata.http.form.*name*.charset
metadata.http.form.*name*.filename

where name corresponds to the data in payload.name.

Simple example

Send a dictionary payload request body which has both key and value strings using the application/x-www-form-urlencoded method:

event HTTPRequestURLEncoding {
   integer id;
   string method;
   string path;
   string contentType;
   dictionary<string, string> data;
}

Send a dictionary payload request body which has a string key and either a string or binary value using the multipart/form-data method; provide the metadata for binary form data using formMetadata:

event HTTPRequestMultiPartForm {
   integer id;
   string method;
   string path;
   string contentType;
   dictionary<string, string> data;
   dictionary<string, dictionary<string,string>> formMetadata;
}

Send a request:

monitor TestFormEncoding {
  action onload() {
    dictionary<string, string> dataURL :=
      {"string":"Hello World", "foo":"bar"};
    dictionary<string, string> dataMultiPart :=
      {"string":"Hello World", "binary": Binary Data};

    //Metadata for form data filed
    dictionary<string,dictionary<string,string>> formMetadata := {
        "binary":{
           "filename":"file.txt",
           "charset":"utf-8",
           "contentType":"text/plain"
         }
    };
    integer id := integer.incrementCounter("HTTPClient.requestId");

    //Using application/x-www-form-urlencoded media type
    send HTTPRequestURLEncoding(id, "POST", "/",
      "application/x-www-form-urlencoded", dataURL) to "http";

    id := integer.incrementCounter("HTTPClient.requestId");
    //Using multipart/form-data media type
    send HTTPRequestMultiPartForm(id, "POST", "/", "multipart/form-data",
      dataMultiPart, formMetadata) to "http";
  }
}

Map the metadata of binary form data using the Mapper codec:

- mapperCodec:
    HTTPRequestMultiPartForm:
      towardsTransport:
        mapFrom:
          - metadata.requestId: payload.id
          - metadata.http.method: payload.method
          - metadata.http.path: payload.path
          - metadata.contentType: payload.contentType
          - metadata.http.form.binary.contentType:
               payload.formMetadata.binary.contentType
          - metadata.http.form.binary.filename:
               payload.formMetadata.binary.filename
          - metadata.http.form.binary.charset:
               payload.formMetadata.binary.charset
          - payload: payload.data

Handling HTML form encoding using a predefined generic event definition

You can invoke an HTTP service with a payload encoded to either multipart/form-data or application/x-www-form-urlencoded media types using the predefined FormRequest event definition. For detailed information about this event definition, see the API reference for EPL (ApamaDoc).

The FormRequest event definition must be used if metadata.contentType is set to either multipart/form-data or application/x-www-form-urlencoded. The request payload must be a dictionary having a string key and string value.

If metadata.contentType is set to application/x-www-form-urlencoded, then the dictionary payload is transmitted as URL-encoded form data.

If metadata.contentType is set to multipart/form-data, then the dictionary payload is encoded to multi-part form data.

Simple example

Use multipart/form-data and application/x-www-form-urlencoded media types with non-ASCII text data form fields:

monitor TestHtmlEncoding
{
  action onload()
  {
    dictionary<string, string> payload := {"foo":"bar", "abc":"def"};

    dictionary<string, dictionary<string, string>> formMetadata :=
        new dictionary<string, dictionary<string, string>>;

    HttpTransport transport :=
        HttpTransport.getOrCreateWithConfigurations("my_host",
        8080, new dictionary<string, string>);
    HttpOptions httpOptions := new HttpOptions;

    // Using application/x-www-form-urlencoded media type
    httpOptions.headers["content-type"] := "application/x-www-form-urlencoded";
    FormRequest(
        transport.createRequest(RequestType.POST, "/", payload, httpOptions),
        formMetadata).execute(handleResponse);

    // Using multipart/form-data media type
    httpOptions.headers["content-type"] := "multipart/form-data";
    FormRequest(
        transport.createRequest(RequestType.POST, "/", payload, httpOptions),
        formMetadata).execute(handleResponse);
  }

  action handleResponse(Response resp)
  {
    log "Got response: " + resp.toString() at INFO;
  }
}

For multipart/form-data, you can still encode binary data form fields. But to do that, you need to develop a custom plug-in which introduces binary data in your customized chain. In that case, the binary data form fields must have the following additional metadata:

• filename
• contentType
• charset

filename is a required parameter. You can provide this metadata as follows:

monitor TestHtmlEncodingBinaryFields
{
  action onload()
  {
    dictionary<string, string> payload :=
        {"foo":"bar", "binary_field":...binary_data};

    dictionary<string, dictionary<string, string>> formMetadata :=   {
      "binary_field":{
     "filename":"file1.txt",
     "charset":"utf-8",
     "contentType":"text/plain"
     }
  };

    HttpTransport transport :=
        HttpTransport.getOrCreateWithConfigurations("my_host",
        8080, new dictionary<string, string>);
    HttpOptions httpOptions := new HttpOptions;

    // Using multipart/form-data media type
    httpOptions.headers["content-type"] := "multipart/form-data";
    FormRequest(transport.createRequest(RequestType.POST, "/", payload,
        httpOptions), formMetadata).execute(handleResponse);
  }

  action handleResponse(Response resp)
  { log "Got response: " + resp.toString() at INFO; }
}

Mapping the body

The HTTP client accepts and returns the payload as a binary object. What the payload consists of depends on the service to which you are connecting. Many services use string-based protocols (such as JSON). For these types of payload, you can use the String codec (see The String codec connectivity plug-in). On messages towards the transport, the String codec takes a string and encodes it in UTF-8 bytes. For messages towards the host, the String codec takes a byte array and decodes it to a string using the UTF-8 encoding. If you are using the String codec, you should put it as the last codec before the HTTP client.

To create a request with no payload (such as a GET request), you should pass an empty string to the String codec, which it will convert to a zero-byte payload. If you are using the “generic” JSON option (see also Using predefined generic event definitions to invoke HTTP services with JSON and string payloads), then you can do the same by sending a new any as the payload. For example:

transport.createRequest(RequestType.GET, "/path", new any, new HttpOptions);

The createGETRequest action will do this for you. In order to recreate this with your own custom chain using the JSON codec, then you need to have an empty payload (which will skip the JSON codec) and then use a second Mapper codec to add an empty string to the payload before the String codec:

- jsonCodec
- mapperCodec:
    "*":
      towardsTransport:
      defaultValue:
      payload: ""
- stringCodec

The resulting string can then be mapped directly into a field in an EPL event, or it can be further processed by other codecs (such as the JSON codec) before the resulting fields are mapped into the Apama event.

If you need to vary your processing depending on the type of the returned data, you may need to write a custom codec in order to handle this. To help with distinguishing different payload types, the HTTP client sets top-level fields to indicate the type of the payload. metadata.contentType contains the MIME type indicated in the Content-Type HTTP header. If present, then metadata.charset indicates the character set from the same HTTP header.

Dealing with cookies

Some HTTP services set cookies and require them to be set in further requests.

When the configuration option cookieJar is true (default), cookies received from the server are stored in memory and added to subsequent outgoing requests. Cookies forwarded using metadata.http.cookies are honored and not overwritten. The HTTP client also honors additional cookie attributes such as path, expiry and max-age. Expired cookies are automatically removed from the internal cache. See also Configuring the HTTP client transport.

When the configuration option cookieJar is false and if you need to take a specific cookie in a response and return it in future requests, you need to map it out into a field in the response event, and then map it back from future request events. The HTTP client stores cookies in metadata.http.cookies.keyname entries. In requests, the HTTP client reads all of the metadata.http.cookies entries and combines them into a single HTTP Cookies header to send to the server. In responses, the HTTP client takes any number of HTTP Set-Cookie headers and turns them into corresponding metadata.http.cookies entries.

Providing HTTP query parameters

HTTP requests can contain request parameters, which are encoded at the end of the URL in the following form:

/path?key=value&key=value

The request parameters can be provided as part of the metadata.http.path element in a request. In this case, however, they must be correctly encoded within the request.

A better solution is to provide the request parameters as part of the metadata.http.queryString element. This is a map of key/value pairs which will be correctly HTTP encoded and appended to the end of the metadata.http.path in the request. The parameters can either be set as a map directly out of the payload, or they can be set individually via the Mapper codec. For example:

- mapperCodec:
    Request:
      towardsTransport:
        mapFrom:
          # set one query parameter individually
          - metadata.http.queryString.param: payload.paramValue
          # alternatively set all query parameters from an EPL dictionary
          - metadata.http.queryString: payload.parameters

Example mapping rules

A full example configuration can be found in the samples directory of your Apama installation. The monitoring sample, found in samples/connectivity_plugin/app/monitoring, can be run both with this pre-compiled HTTP client or with the simple HTTP client sample under samples/connectivity_plugin/cpp/httpclient.

Simple example

The following is a simple REST service with a single URL that is not interested in dealing with error cases:

event PutData {
  integer requestId;
  string requestString;
}
event PutDataResponse {
  integer requestId;
  string responseString;
}

Each PUT request contains a request string which performs an action on the server and returns another string in the response.

startChains:
  simpleRestService:
    - apama.eventMap:
        # Channel that responses are delivered on
        defaultChannel: SRS-response
    - mapperCodec:
        PutData: # requests
          towardsTransport:
            mapFrom:
              - metadata.requestId: payload.requestId
              - payload: payload.requestString
            defaultValue:
              - metadata.http.method: PUT
              - metadata.http.path: /path/to/service
        PutDataResponse:
          towardsHost:
            mapFrom:
              - payload.responseString: payload
              - payload.requestId: metadata.requestId
    - classifierCodec:
        rules:
          - PutDataResponse:
    - stringCodec
    - HTTPClientTransport:
        host: foo.com

CRUD service example

The following is a more complex service that implements a full CRUD (create, read, update, delete) service, with different types of request on different objects. There are several different request types with individual mapping rules. The create request is implemented with these events and mapping rules:

event CreateResource {
   integer id;
   string value;
}
event ResourceCreated {
   integer id;
   string resource;
}

There is one URL for adding new resources which returns the resource identifier which can be used to manipulate it in future via a redirection header.

- mapperCodec:
    CreateResource: # requests
      towardsTransport:
        mapFrom:
          - metadata.requestId: payload.id
          - payload: payload.value
        defaultValue:
          - metadata.http.path: /newResource
          - metadata.http.method: PUT
    ResourceCreated:
      towardsHost:
        mapFrom:
          # redirects us to the new resource
          - payload.resource: metadata.http.headers.location
          - payload.id: metadata.requestId

The full example is provided below:

event GetValue {
   integer id;
   string resource;
}
event CurrentValue {
   integer id;
   string value;
}
event UpdateValue {
   integer id;
   string resource;
   string newValue;
}
event CreateResource {
   integer id;
   string value;
}
event ResourceCreated {
   integer id;
   string resource;
}
event DestroyResource {
   integer id;
   string resource;
}
event ResourceDestroyed {
   integer id;
}
event ResourceNotFound {
   integer id;
   string resource;
}
event InternalError {
   integer id;
   string error;
}

startChains:
  storageService:
    - apama.eventMap:
        # Channel that responses are delivered on
        defaultChannel: storageResponses
    - mapperCodec:
        CreateResource: # requests
          towardsTransport:
            mapFrom:
              - metadata.requestId: payload.id
              - payload: payload.value
            defaultValue:
              - metadata.http.path: /newResource
              - metadata.http.method: PUT
        DestroyResource: # requests
          towardsTransport:
            mapFrom:
              - metadata.requestId: payload.id
              - metadata.path: payload.resource
            defaultValue:
              - metadata.http.method: DELETE
        UpdateValue: # requests
          towardsTransport:
            mapFrom:
              - metadata.requestId: payload.id
              - metadata.path: payload.resource
              - payload: payload.newValue
            defaultValue:
              - metadata.http.method: PUT
        GetValue: # requests
          towardsTransport:
            mapFrom:
              - metadata.requestId: payload.id
              - metadata.path: payload.resource
            defaultValue:
              - metadata.http.method: GET
        ResourceCreated:
          towardsHost:
            mapFrom:
              # redirects us to the new resource
              - payload.resource: metadata.http.headers.location
              - payload.id: metadata.requestId
        ResourceDestroyed:
          towardsHost:
            mapFrom:
              - payload.id: metadata.requestId
        CurrentValue:
          towardsHost:
            mapFrom:
              - payload.value: payload
              - payload.id: metadata.requestId
        ResourceNotFound:
          towardsHost:
            mapFrom:
              - payload.resource: metadata.http.path
              - payload.id: metadata.requestId
        InternalError:
          towardsHost:
            mapFrom:
              - payload.error: metadata.statusReason
              - payload.id: metadata.requestId
    - classifierCodec:
        rules:
          - ResourceCreated:
              - metadata.http.statusCode: 200
              - metadata.http.path: /newResource
          - CurrentValue:
              - metadata.http.statusCode: 200
              - metadata.http.method: GET
          - ResourceDestroyed:
              - metadata.http.statusCode: 200
              - metadata.http.method: DELETE
          - ResourceNotFound:
              - metadata.http.statusCode: 404
          - InternalError:
              - metadata.http.statusCode:
    - stringCodec
    - HTTPClientTransport:
        host: foo.com

Login example

An example with a login request that has to manage cookies might look like this when the service uses JSON:

event Command {
   string command;
   sequence<string> arguments;
}
event Login {
   string username;
   string password;
}
event LoginSuccess {
   dictionary<string, string> sessionCookies;
}
event ExecuteCommand {
   integer id;
   Command command;
   dictionary<string, string> sessionCookies;
}
event CommandResponse {
   integer id;
   string response;
}

The Login command sends a password and sets a cookie which must be set in all the following requests. In practice this may need to be repeated on startup, after some timeout period or certain errors.

startChains:
  remoteAccessService:
    - apama.eventMap:
        # Channel that you send requests to
        subscribeChannels: remoteAccess
        # Channel that responses are delivered on
        defaultChannel: remoteAccess
    - mapperCodec:
        Login: # requests
          towardsTransport:
            mapFrom:
              # payload.user and payload.password will be converted
              # into a JSON document
            defaultValue:
              - metadata.http.path: /login
              - metadata.http.method: PUT
              - metadata.requestId: "" # ignored
        ExecuteCommand: # requests
          towardsTransport:
            mapFrom:
              - metadata.requestId: payload.id
              # set the whole map of any cookies set by the server
              - metadata.http.cookies: payload.sessionCookies
              # a JSON object made from this event
              - payload: payload.command
            defaultValue:
              - metadata.http.method: PUT
              - metadata.path: /execute
        LoginSuccess:
          towardsHost:
            mapFrom:
              # store all cookies set by the server,
              # no matter what they are
              - payload.sessionCookies: metadata.http.cookies
        CommandResponse:
          towardsHost:
            mapFrom:
              # payload.response already parsed from the JSON response
              - payload.id: metadata.requestId
    - classifierCodec:
        rules:
          - LoginSuccess:
              - metadata.http.statusCode: 200
              - metadata.http.cookies.session:
          - CommandResponse:
              - metadata.http.statusCode: 200
    - jsonCodec
    - stringCodec
    - HTTPClientTransport:
        host: foo.com
        tls: true

Content-encoding example

The following example shows how to define content encoding for an HTTP request:

event HTTPRequest
{
    integer id;
    string path;
    string data;
    string method;
    string contentEncoding;
}

You can use the Mapper codec to map the encoding method as follows:

- mapperCodec:
    HTTPRequest:
      towardsTransport:
        mapFrom:
          - metadata.http.path: payload.path
          - metadata.requestId: payload.id
          - metadata.http.method: payload.method
          - metadata.http.headers.content-encoding: payload.contentEncoding
          - payload: payload.data

Monitoring status for the HTTP client

The HTTP client component provides status values via the user status mechanism. It provides the following metrics (where prefix consists of the chain identifier and plug-in name, typically {chainId=HTTPClientChain}.HTTPClientTransport):

For each request/response that is processed, the above MaxInLastHour values are updated if either of the following conditions is true:

• The size of the current message is greater than the existing maximum.
• The existing maximum value was set more than 1 hour ago.

Error responses are not included in the response size metrics. The request size metrics are calculated before compression and the response size metrics are calculated after decompression.

For more information about monitor status information published by the correlator, see Managing and monitoring over REST and Watching correlator runtime status.

Configuring dynamic connections to services

Many applications have a single or small number of statically configured connections to services. For other applications, the connections can be configured dynamically at runtime. To configure the connections dynamically, define your chain under dynamicChains rather than staticChains with the configuration details using dynamic chain replacement variables (@{*varname*}):

dynamicChains:
  HTTPClientChain:
    - apama.eventMap
    *mapping rules...*
    - HTTPClientTransport:
        host: "@{HOST}"
        port: "@{PORT}"

Then you can create instances of that chain configured for specific hosts and ports using the createDynamicChain method on ConnectivityPlugins:

action connectToNewHost(string channelName, string host, integer port,
  string defaultChannelTowardsHost)
    returns Chain
{
     return ConnectivityPlugins.createDynamicChain(
       "http-"+host+":"+port.toString(), [channelName],
       "http", {"HOST":host,"PORT:"port.toString()}, defaultChannelTowardsHost);
}

Events can be sent to the chain via the supplied channelName. When the connection is no longer needed, it can be destroyed via the returned Chain object.

Using predefined generic event definitions to invoke HTTP services with JSON and string payloads

JSON payloads

You can invoke an HTTP service with a JSON payload by using predefined generic Apama event definitions. To do so, you have to use the JSON with generic request/response event definitions option when adding the HTTP client connectivity plug-in. See also Adding the HTTP client connectivity plug-in to a project.

This “generic” option uses a predefined chain definition with dynamic chain instances to invoke multiple HTTP services, and it uses event types in the com.softwareag.connectivity.httpclient package. For detailed information about the available event types, see the API reference for EPL (ApamaDoc).

The following example shows how to invoke an HTTP service using the generic events:

action performRequest() {
    // 1) Get the transport instance
    HttpTransport transport := HttpTransport.getOrCreate("www.example.com", 80);
    // 2) Create the request event
    Request req:= transport.createGETRequest("/geo/");
    // 3) Execute the request and pass the callback action
    req.execute(handleResponse);
}
action handleResponse(Response res) {
    // 4) Handle the response
    if res.isSuccess() {
        // 5) Extract data from the payload
        log res.payload.getString("location.city") at INFO;
    } else {
        log "Failed: " + res.statusMessage at ERROR;
    }
}

Overriding the content-type header of an HTTP request to allow non-JSON string payloads

You can override the content-type header of an HTTP request to allow for non-JSON string payloads.

Whenever the content-type header of a request is not overridden, the payloads are encoded as JSON (this is the default setting). When you override the content-type header, the JSON codec is skipped and the payload is not encoded as JSON, allowing string data to be passed through. The decoding of the response to the request depends on the content type provided by the server.

The following example demonstrates how to override an HTTP request’s content-type header to send string data:

// 1) HTTP PUT request with string ("example string payload") payload
req := transport.createPUTRequest("/plain_string", "example string payload");
// 2) Override the request's content-type header
req.setHeader("content-type", "text/plain");
// 3) Execute the request, passing the callback action handleResponse
req.execute(handleResponse);

Enabling and controlling concurrency

You can create an instance of a transport which uses multiple clients by providing the number of clients when creating it:

HttpTransport transport :=
HttpTransport.getOrCreateWithConfigurations("www.example.com", 80,
{ HttpTransport.CONFIG_NUM_CLIENTS: "3" });

When creating a request, you can specify the concurrency control key or flush behavior for requests that have some dependencies:

Request req:= transport.createGETRequest("/geo/");
req.setConcurrencyControlKey("geo");
req.setConcurrencyControlFlush(true);
req.execute(handleResponse);

For more information on concurrency in the HTTP client, see Executing HTTP requests concurrently.

Restricting the response size

You can create an instance of a transport which limits the response size when creating it:

HttpTransport transport :=
HttpTransport.getOrCreateWithConfigurations("www.example.com", 80,
{ HttpTransport.CONFIG_MAX_RESPONSE_KB: "3", HttpTransport.CONFIG_MAX_RESPONSE_POLICY: "TRUNCATE_END" });

When creating a request, you can limit the maximum response size and the policy to apply:

Request req:= transport.createGETRequest("/logs");
req.setMaxResponseKB(10);
req.setMaxResponsePolicy("REJECT");
req.execute(handleResponse);

For more information on restricting response sizes, see Configuring the HTTP client transport.

Executing HTTP requests concurrently

Enabling concurrency

By default, the HTTP client executes requests serially and in order. As a result, the maximum throughput that can be achieved is limited by the latency of the round-trip to the server. In order to achieve higher throughput if the request processing time cannot be reduced, the HTTP client can start multiple simultaneous connections to the server. These multiple connections can overlap processing of multiple requests, which gives higher throughput.

To enable multiple connections, you must set the numClients configuration option on the HTTP transport. If you are writing your own chain, you would set it as follows in the YAML configuration file:

- HTTPClientTransport:
     host: "hostname"
     numClients: 3

If you are using the generic event definitions, then you can pass the number of clients as an option to getOrCreateWithConfigurations:

HttpTransport transport :=
HttpTransport.getOrCreateWithConfigurations("www.example.com", 80,
{ HttpTransport.CONFIG_NUM_CLIENTS: "3" });

This causes the HTTP transport to create three threads and three persistent connections to the target server. Any idle connection can execute the next request, unless specified otherwise (see below).

Controlling concurrency

Having requests processed concurrently means that responses to those requests may not come back in order. It also means that a later request can begin before an earlier request is complete. This is necessary to get the improved throughput, however, it may be that not all requests are eligible to be processed in any order with respect to each other. For example, two updates to the same entity in a REST API may need to be processed in the correct order to have the correct value afterwards. Alternatively, creating an entity followed by searching for all entities of that type should return the just-created entity.

To allow applications to get the behavior they expect, they can also specify a key on each request called the “concurrency control key”. A concurrency control key serializes all requests with the given key. If set, a request with a concurrency control key waits until any earlier requests with the same key have completed and causes any later requests with the same key to wait until it has completed. This means that only one request can be in-progress at a time for a given concurrency control key. Requests with different keys or no key can be processed concurrently to this request.

For example, in a REST API, the concurrency control key could be set to the item ID when doing a create/read/update/delete on a specific item, to prevent multiple operations on the same item from racing with each other.

In addition, there is a second option called “concurrency control flush” that can be set on each request. This is a Boolean flag that delays this request from starting until all earlier requests have completed (regardless of the concurrency control key). Note that later requests are permitted to start while the flush-enabled request is still executing.

For example, in a REST API, you might set flush on a request that lists or queries multiple items, since you would not want such an operation to start until all earlier create/update/delete operations affecting individual items had completed.

In general, when performing operations on multiple items, flush on the first get after a create/delete/update and vice-versa.

See also the description of the fields metadata.concurrencyControlKey and metadata.concurrencyControlFlush in Mapping events between EPL and HTTP client requests.

These two metadata items can be used together as follows:

• false (or empty, unset, empty string (""), "false" string) evaluates to false.
• Any other value evaluates to true.

For best performance, we recommend one of the following options:

• The number of distinct concurrency control keys used in the system is much larger than the number of clients.
• Or there is a much larger proportion of requests without concurrencyControlKey set than with it.

To set the concurrencyControlKey via the YAML configuration file, you typically need to map it from a field in your event. For example:

- mapperCodec:
    HTTPRequest:
      towardsTransport:
        copyFrom:
          - metadata.concurrencyControlKey: payload.id

Alternatively, if you are using the generic event definitions:

Request req:= transport.createGETRequest("/geo/");
req.setConcurrencyControlKey("geo");
req.setConcurrencyControlFlush(true);
req.execute(handleResponse);

Example of controlling concurrency

Below is an example of when it may be appropriate to use concurrencyControlKey and concurrencyControlFlush in a simplified sequence for demonstration purpose.

This is using the following custom connectivity YAML file with an event MyHTTPRequestWithKeyAndFlush to map to the HTTP request:

startChains:
  http:
    - mapperCodec:
        MyHTTPRequestWithKeyAndFlush:
          towardsTransport:
            mapFrom:
              - metadata.requestId: payload.requestId
              - metadata.http.path: payload.path
              - metadata.http.method: payload.method
              - metadata.concurrencyControlKey: payload.concurrencyControlKey
              - metadata.concurrencyControlFlush: payload.concurrencyControlFlush
              - payload: payload.data

The monitor file used in this example uses a custom event to map to the HTTP request API:

event MyHTTPRequestWithKeyAndFlush
{
   // Standard HTTP request payload fields
   integer requestId;
   string path;
   string data;
   string method;

   // Fields for concurrency control
   any concurrencyControlKey;
   boolean concurrencyControlFlush;
}

You can then proceed as described in the following steps.

Step 1: Create two devices using the device name as the concurrency control key:

send MyHTTPRequestWithKeyAndFlush(0, "/devices/myDevice1", "example data",
  "POST", "myDevice1", false) to chain;
send MyHTTPRequestWithKeyAndFlush(1, "/devices/myDevice2", "example data",
  "POST", "myDevice2", false) to chain;

Step 2: Get myDevice1 information, synchronized on the concurrency control key to ensure that the device has already been created before GET is processed:

send MyHTTPRequestWithKeyAndFlush(2, "/devices", "=myDevice1",
  "GET", "myDevice1", false) to chain;

Step 3: Create two further devices:

send MyHTTPRequestWithKeyAndFlush(3, "/devices/myDevice3", "example data",
  "POST", "myDevice3", false) to chain;
send MyHTTPRequestWithKeyAndFlush(4, "/devices/myDevice4", "example data",
  "POST", "myDevice4", false) to chain;

Step 4: Get information for all existing devices, set concurrency control flush to true to ensure that all prior requests have been processed and recently created devices are returned.

send MyHTTPRequestWithKeyAndFlush(5, "/devices", "=*", "GET", "", true) to chain;

Step 5: Create multiple devices later. This does not depend on anything prior, so no concurrency control flush or concurrency control key is required.

send MyHTTPRequestWithKeyAndFlush(6, "/devices/myDevice[5,6,7,8]",
  "example data", "POST", "", false) to chain;

Step 6: A concurrency control key was not specified in the prior POST, but you may want to act on the result of the next query. Find myDevice5 and then update it. Send with concurrency control flush set to true to ensure that the device is created, and set the concurrency control key to the expected name to ensure that the PUT occurs serially.

send MyHTTPRequestWithKeyAndFlush(7, "/devices", "=myDevice6", "GET",
  "myDevice6", true) to chain;
send MyHTTPRequestWithKeyAndFlush(8, "/devices/myDevice6", "example data", "PUT",
  "myDevice6", false) to chain;

Example of incoming requests

The diagram below shows an example of incoming requests added to the queue as seen on the left, ordered from top to bottom, and one possible example of how the requests may be handled on the three available clients top to bottom.

The numbers represent the order in which the requests were sent.

The solid green line indicates a flush, ensuring all prior events complete before proceeding.

Notice how all requests with the same key are handled in order, requests with a different key are handled concurrently on another client, and requests with no key (empty string "") are handled concurrently on any available client.

The example shown in the above diagram runs as follows:

1. Requests 1 to 12 are sent.
2. Requests 1 and 3 have the same key, therefore they must be handled in order (for example, an updated measurement on a device where key A represents a device A). Requests 2 and 4 can be handled concurrently to request 1 in the meantime, since they do not affect the device A.
3. Request 6 has flush set to true, also indicated by the solid green line. This means that requests 1 to 5 must finish before any further requests can be processed (for example, requesting current measurement values for all devices). Without the flush, there is no guarantee that any of the prior requests have already completed. For example, if the application sent batch 1 to 5, you can query with request 6 whether these return the expected values; without flush, any number of the prior requests might be excluded.
4. Once the queue has flushed, requests 6, 7 and 8 can be processed concurrently.
5. Request 10 has flush set to true and the key set to A. This flushes the queue and ensures that the next request with the key set to A is processed after this one. You may want to update device A, but ensure to have known state before proceeding (for example, that any existing queries have completed).

Monitoring concurrency

The HTTP client publishes the following concurrency-related status items:

• numClients
• serializedRequests
• concurrencyUtilizationPercent

For more information on these metrics, see Monitoring status for the HTTP client.